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An  important  event  for  the  testing  and  acqui¬ 
sition  communities  took  place  Aug.  4  to  8, 
2008.  The  Joint  Test  and  Evaluation  Meth¬ 
odology,  a  chartered  joint  test  and  evaluation 
(T&E)  project,  conducted  a  test  in  conjunction 
with  the  Joint  Battlespace  Dynamic  Deconfliction  event, 
held  by  the  Combined  Test  Organization  for  Future  Combat 
Systems  (FCS).  JBD2  served  as  a  “use  case"  to  evaluate  the 
effectiveness  and  suitability  of  the  JTEM-developed  capabil¬ 
ity  test  methodology. 

JTEM  was  directed  to  develop,  test,  and  evaluate  methods 
and  processes  for  defining  and  using  a  live,  virtual,  and 
constructive  distributed  environment  (LVC-DE)  to  evaluate 
system  of  systems  performance  and  to  understand  system 
contributions  as  they  pertain  to  joint  mission  effectiveness. 
JTEM's  entire  CTM  methods  and  processes,  specifically 
the  CTM  measures  framework,  were  utilized  for  the  first 
time  within  JBD2  to  demonstrate  how  to  accomplish  sys¬ 
tem  of  systems  testing.  This  use  of  the  CTM  allowed  tes¬ 
ters  to  evaluate  system  of  systems  and  the  contribution  of 
specific  systems  through  the  measures  framework  levels  of 
evaluation— joint  mission,  joint  task,  and  system/system  of 
systems  attribute  levels.  It  also  served  as  a  successful  proof 
of  concept  for  the  validation  of  new  systems  and  system  of 
systems  testing  methods  and  processes  that  enable  com¬ 
prehensive  evaluation  of  joint  capabilities. 

The  CTM  is  a  formalization  of  existing  test  processes,  with 
refinement,  for  designing  a  test  of  new  capabilities  or  system 
of  systems  in  a  complex  joint  environment.  The  CTM  is  de¬ 
signed  to  augment,  not  replace,  existing  test  and  evaluation 
methods  and  processes,  and  it  takes  into  account  the  unique 


aspects  of  testing  joint,  networked  systems  in  a  LVC-DE. 
Before  we  discuss  the  test,  some  background  information  is 
needed  to  understand  the  significance  of  the  event. 

Improving  the  Testing  Process 

Over  the  past  several  years,  the  Department  of  Defense  has 
initiated  several  steps  to  make  changes  to  the  testing  and 
acquisition  processes  that  enable  testing  of  joint  capabili¬ 
ties  in  an  operationally  realistic  joint  mission  environment. 
As  DoD  moves  away  from  traditional  single-system  T&E 
approaches  to  the  more  complex  system  of  systems  ap¬ 
proaches,  the  department  must  be  able  to  demonstrate  that 
future  systems  are  fully  integrated,  fully  interoperable,  and 
able  to  meet  the  complexities  and  demands  of  future  bat¬ 
tlespace  environments. 

DoD  testing  and  acquisition  instructions  are  being  reviewed 
to  include  changes  that  declare  the  need  to  test  joint  capa¬ 
bilities  in  a  joint  mission  environment.  The  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  institutes  a 
capabilities-based  approach  to  identifying  current  and  future 
gaps  in  DoD's  ability  to  carry  out  joint  warfighting  missions 
and  functions.  In  support  of  these  policy  directives,  the  DoD 
director  for  operational  test  and  evaluation  developed  the 
Testing  in  a  Joint  Environment  Roadmap  as  a  call  for  action 
to  establish  a  framework  for  life  cycle  evaluation  of  systems 
and  system  of  systems  in  a  joint  mission  environment,  and  to 
institutionalize  evaluation  in  a  joint  operational  context.  The 
roadmap  aims  to  place  testing  in  a  joint  environment  and 
joint  interoperability  testing  at  the  core  of  T&E  activity  while 
promoting  changes  in  how  DoD  does  business  in  the  areas 
of  policy,  infrastructure,  and  methods  and  processes.  Policy 
changes  are  being  driven  through  the  roadmap  governing 
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As  testing  in  the  joint 
environment  continues  to  require 
more  of  the  T&E  community, 
the  capability  test  methodology 
will  ensure  that  we  test  with 
relevance. 


body  and  through  established  DoD  processes.  Infrastruc¬ 
ture  networks  and  middleware  issues  are  being  addressed 
through  the  Joint  Mission  Environment  Test  Capability 
(JMETC)  program  under  the  Test  Resource  Management 
Center,  a  field  activity  reporting  to  the  Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logis¬ 
tics.  The  JTEM  joint  test  and  evaluation  project,  chartered 
by  the  DoD  director  for  operational  test  and  evaluation  in 
February  2006,  is  addressing  methods  and  processes  for 
evaluating  capabilities  in  the  joint  environment.  The  August 
2008  test  event  was  part  of  the  JTEM  project. 

Developing  CTM  Version  3.0 
For  the  past  two  years,  JTEM  developed,  tested,  and  evalu¬ 
ated  methods  and  processes  for  defining  and  using  an  LVC- 
DE  to  evaluate  system  of  systems  performance  and  joint 
mission  effectiveness.  Before  August  2008,  JTEM  con¬ 
ducted  rock  drills,  a  gap  analysis,  and  a  test  event  to  gather 


feedback  on  its  CTM  development,  leading  to  the  release  of 
CTM  version  2.0  for  the  JBD2  test  event.  The  FCS  Combined 
Test  Organization  planned  the  JBD2  event  to  establish  a  rig¬ 
orous  test  context  that  would  allow  the  test  participants  to 
examine  FCS  test  technology  requirements  needed  for  test¬ 
ing  in  a  joint  environment  in  support  of  acquisition  milestone 
test  activities.  JTEM  leveraged  the  event  to  test  and  evaluate 
CTM  methods  and  processes  when  used  by  a  typical  test  or¬ 
ganization  under  operationally  realistic  conditions.  For  JTEM, 
the  test  article  was  the  CTM.  Results  gathered  throughout 
the  planning,  execution,  and  post-execution  phases  of  JBD2- 
provided  areas  for  CTM  improvement  to  be  incorporated 
into  CTM  version  3.0,  scheduled  for  release  at  the  conclu¬ 
sion  of  the  JTEM  joint  T&E  project  in  April  2009. 

Mutual  Benefits 

FCS  and  JTEM  formed  a  natural  partnership  for  the  August 
2008  test  event.  FCS  leaders  recognized  the  event  pre¬ 
sented  challenges  requiring  new  testing  capabilities  (system 
of  systems  on  a  large  scale;  move,  shoot,  and  communicate 
simultaneously;  seamless  integration  with  joint  elements 
for  network-centric  operations).  To  accommodate  this  new 
complexity,  DoD  strategic  planning  guidance  demanded  cre¬ 
ation  of  a  joint  environment  testing  capability.  As  pointed 
out  earlier,  this  DoD-level  requirement  led  to  the  creation 
of  JMETC  and  JTEM  as  part  of  the  larger  testing  in  a  joint 
environment  initiative.  As  the  FCS  testing  strategy  matured 
along  with  JTEM,  a  mutually  beneficial  relationship  emerged 
and  grew  into  the  JBD2  partnership.  JTEM  understood  JBD2 
would  provide  the  opportunity  to: 

■  Identify  issues  in  integrating  the  end-to-end  CTM  into 
existing  test  activities 

■  Develop  and  mature  the  LVC-DE 

■  Investigate  data  requirements  issues 

■  Identify  deficiencies  in  implementation  of  the  live,  vir¬ 
tual,  and  constructive  joint  mission  environment. 
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FCS  saw  the  partnership  with  JTEM  as  a  way  to  establish  a 
rigorous  test  context  in  which  to  examine  FCS  test  technol¬ 
ogy  requirements  needed  for  testing  in  a  joint  environment 
in  support  of  Milestone  C  test  activities,  as  well  as  enable 
risk  reduction  for  critical  FCS  technology  areas.  Equally 
important,  JMETC  used  JBD2  as  an  opportunity  to  charac¬ 
terize  the  network  infrastructure  and  mature  the  baseline 
capability  required  to  support  system  of  systems-level 
testing  across  the  DoD  components.  Overall,  JBD2  pro¬ 
vided  a  requirement  for  a  high-fidelity,  real-time,  rapidly 
configurable,  distributed  network  connecting  virtual  and 
constructive  models  with  live  systems.  This  architecture 
offered  the  participating  organizations  an  opportunity  to 
test  selected  initiatives  in  an  operational  environment, 
enhancing  the  overall  joint  context  of  the  test  while  al¬ 
lowing  stakeholders  to  investigate  these  important  new 
technologies.  At  the  same  time,  participation  in  the  test 
event  provided  the  JTEM  test  team  close-in  visibility  on 
implementation  of  the  CTM  in  a  joint  mission  environ¬ 
ment.  The  overarching  test  goals  were: 

■  Evaluate  the  effectiveness  and  suitability  of  the  JTEM 
CTM 

■  Test  the  FCS  network,  technologies,  and  distributed 
environment  for  use  in  future  tests  for  acquisition  mile¬ 
stone  decisions 

■  Mature  the  JMETC  baseline  capability  to  support  sys¬ 
tem  of  systems-level  testing  across  the  Services. 

A  Complex  Test  Scenario 

To  achieve  these  goals,  JBD2  established  a  complex  joint 
mission  environment  composed  of  16  test  sites  and  more 
than  40  unique  live,  virtual,  and  constructive  systems  con¬ 
nected  across  four  time  zones.  These  test  sites  represented 
all  four  Services  and  the  U.S.  Joint  Forces  Command.  Of 
these  16  JBD2  sites,  10  were  reused  from  two  previous  test 
venues  that  were  a  part  of  a  series  of  events  culminating 
in  JBD2.  Seven  Service  and  joint  initiatives  were  included 
as  part  of  the  test  architecture.  JBD2  truly  provided  joint 
context  and  stressed  the  boundaries  of  a  live,  virtual,  and 
constructive  joint  mission  environment. 

The  JBD2  test  scenario  focused  on  the  complex  and  dynamic 
battlespace  deconfliction  problem.  The  joint  mission  envi¬ 
ronment  centered  on  this  problem,  with  mission  tasks  se¬ 
lected  that  would  require  complete  coordination  of  the  Ser¬ 
vices  to  execute.  Four  unique  Joint  Fires  and  two  Joint  Close 
Air  Support  key  mission  tasks  were  agreed  upon  in  order  to 
fully  stress  the  notional  joint  air  ground  system  of  systems 
(JAGS)  under  test.  The  scenario  design  called  for  these 
mission  tasks  to  occur  simultaneously  and/or  sequentially 
as  the  battle  unfolded  alongside  organic  fires-and-effects 
missions  and  air  operations.  Test  design  factorials  incorpo¬ 
rating  materiel  and  non-materiel  solutions  were  developed 
that  required  four  unique  test  configurations  for  every  key 
mission  task.  A  “free  play"  environment  was  established  in 
which  any  number  of  key  mission  task  types  could  execute 
in  parallel  for  a  single  test  configuration. 


DoD  testing  and  acquisition 
instructions  are  being  reviewed 
to  include  changes  that  declare 
the  need  to  test  joint  capabilities 
in  a  joint  mission  environment. 


As  the  JBD2  test  scenario  unfolded  and  the  key  mission  tasks 
were  executed,  JTEM's  CTM  measures  framework  was  im¬ 
plemented,  which  enabled  critical  test  data  to  be  collected. 
The  data  permitted  evaluation  of  the  notional  JAGS  not  only 
at  the  attribute  level  but  also  at  the  joint  task,  joint  mission 
desired  effect,  and  joint  mission  end  state/objectives  levels, 
thus  effectively  describing  whether  or  not  JAGS,  as  a  joint 
capability,  provided  the  means  and  ways  to  perform  a  set 
of  tasks  to  achieve  the  set  of  desired  effects  that  lead  to 
mission  success.  However,  for  JBD2  the  notional  JAGS  test 
data  was  not  the  JTEM  focus.  JTEM's  focus  was  on  the  ef¬ 
fectiveness  and  suitability  of  the  CTM  to  collect  the  right 
data  and  evaluate  it  up  to  the  mission  measure  of  effective¬ 
ness  level. 

A  Successful  Test  Event 

JBD2  was  a  major  success  in  several  ways.  JTEM's  CTM  was 
fully  exercised  within  a  test  environment  possessing  a  level 
of  complexity  not  seen  in  previous  test  events.  The  number 
of  test  sites  connected,  along  with  the  number  of  systems 
represented,  is  extraordinary  given  the  planning  process  was 
completed  in  less  than  one  year.  Additionally,  the  level  of 
effort  and  use  of  the  CTM  were  significant  given  the  test 
team's  short  timeframe  for  planning  and  executing  the 
test  event.  Reusability  of  previous  JTEM  test  products  and 
processes— Joint  Fires  mission  threads,  the  JMETC  virtual 
private  network,  DoD  Architecture  Framework  products, 
and  CTM  products— exceeded  expectations.  Data  collection 
and  reduction  were  conducted  at  a  speed,  accuracy,  and 
fidelity  that  set  new  standards  for  distributed  testing.  Eighty 
percent  of  the  measures  framework  metrics  was  collected 
and  reduced  within  a  week  of  the  test  event.  Data  for  all  key 
mission  task  measures  of  effectiveness,  task  measures  of 
performance,  and  measures  of  system  of  systems  attributes 
were  collected  and  reduced  within  a  month  of  test  execution 
completion. 

JBD2  was  clearly  a  successful  proof  of  concept  in  collect¬ 
ing  data  and  evaluating  mission  measures  of  effectiveness, 
according  to  JTEM's  CTM  measures  framework.  The  data 
showed  the  impact  of  system  of  systems  configurations  on 
mission-desired  effect  and  joint  mission  effectiveness.  This 
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is  a  significant  milestone  because  testers  can  now  move  for¬ 
ward  in  their  attempts  to  institutionalize  testing  in  a  joint 
mission  environment  across  DoD. 

From  JTEM's  perspective,  JBD2  execution  of  CTM  prod¬ 
ucts  was  more  than  sufficient  to  assess  the  effectiveness 
and  suitability  of  the  CTM.  The  significant  lessons  learned 
from  the  test  event  were  of  great  value  to  the  JTEM  test 
team  as  the  team  enhances  its  methods  and  processes  and 
incorporates  improvements  into  CTM  version  3.0.  About 
two-thirds  of  the  CTM  products  produced  during  the  test 
aligned  completely  or  partially  with  the  CTM  guides  and 
model  descriptions.  Additionally,  the  JBD2  test  team  devel¬ 
oped  several  test  products  that  were  outside  the  domain  of 
the  CTM.  JTEM  is  carefully  considering  these  test  products 
for  inclusion  in  CTM  version  3.0. 

Insights  on  Improvement 

JBD2  provided  several  insights  that  the  test  team  could  use 
in  CTM  development  or  in  recommendations  for  improve¬ 
ments  to  current  test  and  acquisition  practices.  A  high-level 
summary  of  these  insights  finds: 

■  More  focus  should  be  spent  on  defining  and  document¬ 
ing  a  coherent  evaluation  strategy  and  joint  mission 
effectiveness  requirements.  A  preponderance  of  early 
test  activities  and  resources  are  devoted  to  environ¬ 
ment  design  and  build.  Focusing  first  on  the  evaluation 
strategy  will  reduce  the  expenditure  of  resources  at 
execution  time. 

■  There  is  a  need  for  a  standard  lexicon  for  conducting 
T&E  in  an  LVC-DE. 

■  JCIDS  products  are  not  yet  sufficiently  available  to  ef¬ 
fectively  create  the  joint  operational  context  for  a  test. 

■  There  is  a  need  for  data  standardization  and  improved 
data  access  and  retrieval  methods. 

■  There  is  a  need  for  enterprise-level  support,  expertise, 
and  coordination  in  order  to  effectively  test  in  a  joint 
mission  environment. 

CTM  Handbooks 

Along  with  these  insights,  more  detailed  and  specific  im¬ 
provements  to  the  CTM  have  been  captured  by  the  JTEM 
team  as  a  result  of  CTM  usage  in  JBD2.  These  improvements 
will  be  included  in  CTM  version  3.0,  which  will  contain  a 
CTM  disc  with  navigation  slide  and  three  publications: 

■  The  Action  Officer  Handbook  for  Testing  in  a  Joint  Environ¬ 
ment 

•  The  Program  Management  Handbook  for  Testing  in  a  Joint 
Environment 

•  The  Joint  Mission  Effectiveness  Analysis  Handbook 
These  publications  incorporate  CTM  guides  that  provide  a 
roadmap  and  context  to  methods  and  processes  for  testing 
in  a  joint  environment. 

The  Action  Officer  Handbook,  written  for  typical  test  and  event 
planning  staffs,  will  address  concepts  supporting  testing  in  a 
joint  environment  and  will  contain  comprehensive  CTM  user 


guides  for  each  step  of  the  CTM  process,  printable  checklist 
extracts  to  be  used  along  with  the  CTM  guides,  and  annexes 
containing  useful  information  such  as  the  CTM  lexicon.  The 
Program  Management  Handbook  will  be  an  executive-level 
publication  providing  an  enterprise  view  of  DoD  acquisition 
transformation,  addressing  testing  in  a  joint  environment, 
and  offering  an  overview  of  the  CTM  for  program  and  capa¬ 
bility  portfolio  managers.  The  Analysis  Handbook ,  written  for 
test  planners  and  test  analysts,  will  provide  the  framework 
and  supporting  information  for  the  CTM  evaluation  thread 
process  and  will  offer  recommended  analysis  tools  and  tech¬ 
niques  as  well  as  annexes  with  supporting  information  and 
hyperlinks  between  sections  for  online  use.  That  CTM  pack¬ 
age  will  prepare  the  program  manager  and  test  organization 
staff  to  effectively  test  as  the  capabilities-based  approach 
to  acquisition  requires. 

Promising  Results 

JBD2  provided  the  opportunity  to  identify  issues  in  integrat¬ 
ing  the  end-to-end  CTM  into  an  existing  test  activity,  and  it 
allowed  the  T&E  community  to  gain  a  better  understanding 
of  what  is  required  to  fully  realize  a  sufficient  capability  to 
test  in  a  joint  mission  environment.  The  results  are  promising 
and  indicate  the  CTM: 

■  Reduces  cycle  time  for  capability  development  and  test¬ 
ing 

■  Increases  the  speed  of  data  collection,  reduction,  analy¬ 
sis,  and  evaluation 

■  Integrates  developmental  and  operational  testing 

■  Provides  a  needed  operational  evaluation  framework 
from  which  to  test  system  of  systems  in  a  joint  mission 
environment. 

There  is  still  more  work  to  be  done.  In  the  near  future,  CTM 
version  3.0  must  be  integrated  into  existing  test  processes 
within  the  acquisition  community  in  a  real  test  event  to  in¬ 
form  an  acquisition  decision.  This  will  further  validate  the 
CTM  and  provide  valuable  lessons  learned  for  its  improve¬ 
ment.  This  action  will  also  serve  to  operationalize  the  CTM, 
providing  a  springboard  for  the  CTM  to  be  institutionalized 
as  the  methods  and  processes  of  choice  for  testing  in  a  joint 
mission  environment. 

The  CTM  promises  to  improve  how  we  test  and  acquire  joint 
capabilities.  This  promise  will  not  be  fulfilled  unless  we  all 
support  the  CTM  as  a  means  of  providing  improved  T&E  of 
joint  capabilities.  As  testing  in  the  joint  environment  con¬ 
tinues  to  provide  more  challenges  for  the  T&E  community, 
the  CTM  will  ensure  that  we  test  with  relevance,  thus  sup¬ 
porting  the  acquisition  and  delivery  of  "plug  and  fight"  joint 
warfighting  capabilities. 


The  authors  welcome  questions  and  comments  and  can  be 
contacted  at  steven.hutchison@disa.mil  max.lorenzo@jte. 
osd.mil,  and  daniel.bryan@jte.osd.mil. 
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